Audiovisual Perception of Spoken Words in Speech and Nonspeech Modes: Measures of Architecture and Capacity

Contemporary models of audiovisual speech perception attempt to explain accuracy data based on curve fitting and optimization techniques (see Braida, 1991; Massaro, 1987). Research on audiovisual speech perception lacks a formal mathematical foundation because current models do not make predictions about reaction time or adequately describe how the audio and visual channels are processed in the black box. The double factorial paradigm (DFP) developed by Townsend and Nozawa (1995) uses systems factorial technology to provide a framework for investigating how different information channels are processed. In Experiment 1, participants were required to make one response if auditory information, visual information, or both forms of information were present and a negative response on target absent trials. Data from the audiovisual detection task with the word “base” as the stimulus showed that processing architecture was either coactive or parallel self-terminating. A second experiment again using the double factorial paradigm methodology (Experiment 2) required participants to distinguish between two spoken words: “base” and “face.” The data showed that processing was mostly coactive, but possibly parallel self-terminating in some cases. Processing capacity was limited in both experiments, indicating a lack of redundancy gain. Overall, these results suggest that the audio and visual channels are combined into a single processor, although inhibition or competition may exist between channels.